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I.  Introduction 


The  purpose  of  this  research  is  to  develop  minimally  invasive  procedural  simulation 
technologies  that  are  broadly  applicable  to  the  improvement  of  medical  training.  The 
implementation  of  this  technology  will  enable  the  Department  of  Defense  (DoD)  to 
provide  improved  medical  support  to  the  wounded  soldier  through  enhanced  medical 
training  with  improved  diagnosis,  rehearsal,  and  treatment  planning.  This  project 
leverages  the  involvement  of  ongoing  DoD  and  civilian  work  in  virtual  reality  and  will 
improve  military  readiness  through  shortened  recovery  times.  It  will  also  lead  to  the 
production  of  commercially  viable  products  with  educational  and  training  benefits  for 
U.S.  hospitals  and  medical  schools.  The  specific  application  chosen  for  this  research  was 
the  simulation  of  flexible  ureteroscopy. 


II.  Body 

Overview  of  Ureteroscopy 

A  task  analysis  of  ureteroscopy  was  performed.  A  flow  diagram  of  this  task  analysis  is  in 
Appendix  A.  Briefly,  however,  ureteroscopy  is  the  placement  of  an  endoscope  into  the 
ureter  and  up  to  the  kidney.  Access  to  the  ureter  is  obtained  by  inserting  a  rigid 
cystoscope  through  the  urethra  and  into  the  bladder.  A  guidewire  is  then  navigated 
through  the  ureterovesical  junction.  A  balloon  catheter  is  used  to  dilate  the  distal  portion 
of  the  ureter  where  it  inserts  in  the  bladder.  Occasionally  the  proximal  portion  of  the 
ureter  must  also  be  dilated.  Illustrations  of  the  pertinent  anatomy  are  shown  in  Figure  1 
of  Appendix  B. 

The  ureteroscope  is  navigated  through  the  ureter  using  both  the  endoscopic  and 
fluoroscopic  views.  Both  views  are  essential  to  this  procedure.  Contrast  is  used  to 
ascertain  the  location  and  size  of  the  ureteral  lumen.  The  scope  may  be  advanced  up  to 
the  kidney.  Tools  may  be  placed  through  the  working  channel  to  perform  a  variety  of 
procedures. 

Overview  of  Technical  Accomplishments 

The  overall  technical  product  of  this  work  is  an  initial  version  of  a  ureteroscopy  training 
simulator.  The  simulator  consists  of  hardware,  software,  and  data  elements. 

The  COTS  hardware  consists  of  a  550MHz  Pentium  III  computer  with  128M  RAM 
equipped  with  an  OpenGL  accelerated  graphics  card  (a  WinTel  computer  with  a  GeForce 
display  card).  The  custom  hardware  consists  of  an  interface  device  that  simulates  male 
urological  anatomy  along  with  a  modified  Storz™  ureteroscope.  We  modified  to  enable 
detection  of  motion  of  the  thumb  lever  to  detect  articulation  of  the  scope  tip.  We 
experimented  with  an  instrumented  working  channel  with  force  feedback  on  the  working 
channel  tool  that  we  previously  implemented  in  a  bronchoscope,  but  we  did  not  complete 
development  of  a  sufficiently  small  version  to  fit  within  the  confines  of  the  ureteroscope. 

COTS  software  consists  of  the  Microsoft  Windows  NT  4.0  Operating  System  software, 
Blue  Water  Systems  real  time  kernel,  and  various  drivers  for  OpenGL  and  hardware 
interface  boards.  Custom  software  includes  the  integrated  simulation  system  with 
modules  for  hardware  interface,  physiological  and  anatomical  modeling,  lighting, 
kinematics  and  dynamics,  fluid  flow  and  particle  dynamics.  Custom  developed  models 
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(data)  include  those  of  the  bladder,  the  ureter,  the  calyces  of  the  kidney,  surface  textures, 
and  models  for  interaction  of  fluids  with  lighting  and  particles. 

The  integrated  system  achieves  a  minimum  real-time  frame  rate  of  20  frames  per  second. 
The  system  was  evaluated  during  development  and  on  completion  by  engineers, 
physicians,  and  training  specialists  to  gauge  suitability  for  training  use  and  to  determine 
whether  solutions  to  stability,  anatomical  modeling  and  interaction  modeling  problems 
were  satisfactory. 

Images  of  the  prototype  simulator  developed  under  this  contract  are  shown  in  Figures  1-3 
in  Appendix  C.  Additional  detail  of  our  accomplishments  and  positive  and  negative 
findings  under  each  technical  task  in  the  approved  statement  of  work  are  provided  in  the 
sections  below. 

Techniques  for  the  simulation  of  physiological  events 

We  developed  models  of  anatomical  motion  caused  by  scope-anatomy  interaction,  as 
well  as  interaction  with  other  nearby  anatomy.  In  particular,  the  renal  artery  causes  very 
visible  motion  which  is  useful  as  a  landmark  during  navigation  of  the  ureter.  We 
implemented  a  system  that  utilizes  moving  line  segments  outside  the  visible  model  to 
simulate  such  phenomena.  We  couple  motion  of  these  external  lines  with  a  number  of 
parameterized  variables  for  distance  and  coupling  compliance  to  provide  easily  adjustable 
and  realistic  coupling  to  the  anatomical  model. 

We  implemented  a  real-time  fluoro  display  that  models  fluid  flow  and  diffusion.  In  order 
to  obtain  real-time  response,  we  chose  to  model  fluid  flow  at  the  surface  of  the 
anatomical  model  thus  yielding  a  two  dimensional  (surface)  representation  of  the  three- 
dimensional  (volume)  phenomenon.  The  model  is  linked  with  the  actual  polygons  of  the 
inner  surface  of  the  ureter  and  the  calyces  of  the  kidney.  The  method  works  well  with 
moving  anatomy  and  thus  enables  realistic  synchronization  of  endoscopic  and  fluoro 
views. 

The  resultant  reduction  of  dimensionality  greatly  speeds  the  fluoro  view  computations. 
Our  initial  examination  of  the  implications  of  rendering  a  surface  vs.  a  volume  suggested 
that  the  opacity  of  the  resultant  image  would  be  higher  at  the  edges  than  the  center  of  the 
lumen,  whereas  the  actual  view  would  be  more  opaque  near  the  center.  In  evaluations 
with  urologists,  we  found  that  the  differences  between  our  technique  and  actual  fluoro 
were  minimal  and  were  not  judged  to  be  unrealistic. 

We  plan  in  the  future  to  extend  the  model  to  accurately  reflect  interactions  encountered  in 
complications  such  as  dissection  of  the  lining  of  the  ureter. 

Creation  of  realistic  endoscopic  lighting  simulation 

The  endoscopic  view  through  a  ureteroscope  is  made  complex  by  a  number  of 
phenomena.  Saline  solution  and  urine  have  different  refractive  indices  and  different 
coloration.  Lighting  projected  from  the  scope  tip  reflects  off  the  bladder  and  ureter  walls 
and  produces  specular  reflections  whose  shapes  point  in  the  direction  of  curvature. 
Kidney  stones  have  a  characteristic  surface  roughness,  and  when  fragmented  for  removal, 
can  produce  a  cloud  of  small  particles,  all  of  which  swirl  around  in  the  field  of  view  with 
the  fluid  flow  from  the  endoscope.  Representing  these  elements  realistically  could  easily 
require  a  supercomputer  for  adequate  real  time  rendering.  Our  target  platform  was  a 
550MHz  PHI  processor  and  a  hardware  accelerated  OpenGL  graphics  card.  Our 


HT  Medical  Systems,  Inc. 


Final  Report 


Page  5 


processing  budget  for  setting  up  the  graphics  pipeline  and  rendering  is  30  ms 
(milliseconds),  based  on  a  total  67  ms  overall  frame  time.  We  therefore  experimented 
with  a  number  of  effects  that  provide  a  realistic  visual  simulation  while  still  being  able  to 
meet  the  rendering  budget. 

To  simulate  the  specular  reflection  pointing  towards  the  center  of  the  lumen  of  the  ureter, 
we  use  environment  maps  which  we  move  in  realtime  in  concert  with  a  reduced 
dimensionality  model  of  the  scope  and  ureter.  We  modified  methods  previously 
developed  for  representation  of  mucus  on  endoscope  lenses  to  produce  effects  that  look 
realistically  like  urine  and  saline  mixing.  While  the  visual  effect  is  judged  to  be  good, 
real  time  control  of  the  image  requires  a  complex  fluid  flow  model  which  we  only  were 
able  to  partially  implement.  Our  final  design  involved  a  simplified  flow  model  with  a 
steady  stream  out  the  end  of  the  scope  that  moves  in  a  toroidal  pattern  -  an  elongated 
smoke-ring.  The  visual  effects  produced  by  this  technique  produce  slight  where  the  flow 
interacts  with  the  anatomy.  We  have  redesigned  the  system  to  incorporate  a  flow  pattern 
that  is  warped  by  a  set  of  control  points  that  are  in  turn  controlled  by  a  low-order  particle 
system  but  were  not  able  to  complete  implementation  due  to  resource  constraints.  We 
will  plan  to  complete  implementation  in  the  future. 

Real-time  collision  processing 

The  complexity  of  the  anatomy  of  the  urinary  system  requires  a  large  number  of  polygons 
to  represent  realistically.  The  addition  of  dynamically  movable  particles,  such  as  kidney 
stones  greatly  increases  the  complexity  of  the  task.  We  have  used  a  number  of 
techniques  to  make  the  collision  detection  processing  manageable.  We  break  the 
anatomy  into  a  series  of  viewing/collision  groups  that  are  at  successively  greater 
distances  from  the  point  of  insertion.  We  use  these  groups  as  a  coarse  filter  for  the 
collision  detection  algorithm.  Within  the  collision  groups  we  use  oriented  bounding 
boxes  (OBB)  for  objects  in  an  object-to-object  collision  detection  algorithm.  This 
intermediate  filter  minimizes  the  number  of  potential  collisions  passed  to  the  low  level 
collision  detection  processing,  where  point  to  polygon  collision  detection  processing  is 
performed.  We  take  advantage  of  the  minimal  motion  of  most  areas  of  the  anatomy,  and 
combining  this  with  the  temporal  coherence  of  scope  movement,  we  have  been  able  to 
achieve  our  target  20  ms  collision  detection  processing  in  most  areas  of  the  model. 

We  developed  an  overall  structure  to  realistically  handle  interactions  with  kidney  stones. 
These  present  a  number  of  challenges  that  required  extension  of  the  framework. 
Interaction  of  the  scope  and  working-channel  tools  with  these  objects  that  are  in  the 
anatomy  but  not  attached,  and  can  tumble  and  move  freely  required  a  specialized 
collision  detection  model.  Fragmentation  of  stones  in  real-time  required  a  structure  for 
model  representation  which  is  more  dynamic  than  other  parts  of  the  anatomy.  We 
developed  stones  that  react  with  each  other  and  the  anatomy,  and  that  can  be  fragmented 
in  arbitrary  ways  while  retaining  the  energy  and  momentum  of  the  original  stone  in  the 
resulting  particles.  We  still  have  some  problems  with  stone-model  interpenetration,  but 
the  basic  framework  is  in  place  and  works.  Stones  currently  do  not  interact  with  tools  or 
the  scope  and  we  will  incorporate  this  in  the  future. 

Representations  of  the  physics  of  the  flexible  endoscope 

The  problem  of  simulating  the  physics  of  the  flexible  endoscope  is  complicated  by  the 
interaction  of  the  scope  with  instruments  placed  through  the  working  channel  of  the 
scope.  During  initial  navigation,  we  assume  a  navigational  wire  has  already  been  placed 
in  the  ureter.  The  scope  is  then  threaded  over  the  navigational  wire  through  the  urethra 
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into  the  bladder  and  then  into  the  ureter.  We  have  created  a  combination  physics  model 
for  the  scope  based  on  a  rigid  body  representation  of  the  individual  segments  of  the 
scope.  We  use  penalty  forces  to  represent  forces  from  the  anatomy  on  the  scope  and  then 
solve  a  set  of  ordinary  differential  equations  using  a  second  or  forth  order  Runge-Kutta 
method.  We  compute  corrective  forces  on  the  rigid  body  links  to  enforce  rigid  length 
constraints  on  the  scope  elements,  then  move  forward  one  time  step.  For  stability  and 
realism  we  limit  both  acceleration  and  velocity.  At  each  time  step,  we  compare  the 
dynamic  solution  with  a  static  equilibrium  solution  computed  using  the  principle  of 
virtual  work. 

We  had  stability  problems  with  the  fluoroscopic  view  caused  by  high  frequency  artifacts 
in  the  solution  of  the  physics  equations  of  motion  for  the  ureteroscope.  The  real-time 
rendering  of  the  scope  on  fluoro  was  adequate  but  the  rendering  of  the  scope  view  was 
too  jittery.  We  used  a  parameterized  Kalman  filter  process  to  remove  unwanted  high 
frequency  components  while  retaining  realistic  responsiveness  when  the  scope  is 
manipulated  quickly 

We  have  constructed  a  number  of  alternative  physics  based  models  of  general  purpose 
endoscopes  and  evaluated  them  for  stability  across  the  range  of  input  motions  that  are 
anticipated  in  procedures  of  this  type.  We  refined  the  modeling  system  and  to  enable 
interactively  changing  the  physical  properties  of  the  scope  and  navigational  wire  in  real¬ 
time,  allowing  simulation  of  the  insertion  and  withdrawal  of  working  channel  tools  into 
the  scope  during  navigation  with  the  resultant  decrease  in  navigational  flexibility. 

We  devised  a  two  level  deformation  and  interaction  model  to  obtain  adequate  speed  and 
responsiveness.  A  global  model  is  used  to  quickly  compute  macro  deformation  of  the 
bladder  and  ureter,  and  to  compute  the  forces  transmitted  by  the  scope  back  to  the 
practitioner,  and  a  micro  model  is  used  for  computation  and  display  of  the  localized, 
small  deformations  of  the  anatomy  caused  by  the  tip  of  the  scope  or  tools  inserted 
through  the  working  channel.  We  are  using  an  innovative  application  of  dynamic  B- 
Splines  that  permits  adaptable  inertia,  elasticity,  and  damping  parameters  to  be  applied  on 
a  local  or  global  basis  in  response  to  modeled  physiological  effects. 

Integration  of  a  computer-based  geometric  model  of  the  urinary  tract 
We  have  developed  a  model  of  the  urinary  tract  including  the  urethra,  the  bladder,  the 
ureter,  and  the  calyces  of  the  kidney.  We  developed  the  model  using  a  nurbs-based 
modeling  tool,  then  used  a  custom  tool  to  produce  a  polygonal  model  that  has  the 
capability  of  deforming  with  tool/anatomy  interaction.  We  augment  the  resultant  data 
structure  with  a  number  of  additional  indices  that  are  implemented  as  threaded  lists  of 
pointers  that  enable  traversal  of  the  model  in  a  non-linear  fashion.  These  are  produced 
with  an  off-line  utility  and  enable  us  to  quickly  access  neighboring  polygons  and  vertices 
to  speed  up  real-time  collision  detection  between  the  scope,  the  wire,  and  the  anatomy 

To  aid  in  recognition  of  anatomical  landmarks,  we  created  a  generic  framework  for 
OpenGL  billboards,  which  are  two-dimensional  elements  that  exist  in  three  dimensions 
by  automatically  rotating  to  face  the  camera  position.  These  are  used  to  add  “road  signs” 
that,  when  enabled,  provide  labels  for  anatomical  structures  that  pop  up  as  the  user 
explores  the  anatomy.  These  aid  in  learning  the  basic  anatomy  and,  along  with  the  fluoro 
and  external  views,  assist  the  student  in  developing  a  good  understanding  of  the 
relationships  between  scope  manipulation  and  scope  orientation  in  the  anatomy. 
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Tactile/force  feedback 

We  designed  a  number  of  testbeds  for  evaluation  of  alternative  means  of  adapting  our 
haptic  interface  devices  to  enable  the  application  of  a  range  of  working  channel  tools  with 
realistic  sensations  of  touch  and  feel  on  both  the  scope  and  the  working  channel 
simultaneously.  We  implemented  a  mechanism  for  providing  force  feedback  from  within 
the  body  of  the  scope  itself  and  evaluated  the  efficacy  of  that  mechanism  in  comparison 
with  application  of  forces  at  the  tip  of  the  working  channel  tool.  Of  particular  concern  is 
the  ability  to  couple  the  forces  displayed  on  the  two  devices  with  motions  of  the  devices 
relative  to  each  other.  We  developed  and  evaluated  a  two  level  haptic  feedback  system 
which  combines  a  1000  Hz  primary  feedback  loop  implemented  in  an  external 
microprocessor  with  a  100  Hz  global  feedback  loop  implemented  in  the  simulation 
computer.  This  combination  was  intended  to  allow  us  to  realistically  and  accurately 
display  complex  forces  such  as  those  associated  with  the  use  of  a  basket  while 
simultaneously  manipulating  both  the  scope  and  the  basket. 

In  our  evaluation  of  this  two  processor  approach,  we  found  that  the  force  feedback 
mechanism  for  the  wire  is  highly  responsive.  However,  we  have  had  continuing 
problems  with  getting  sufficient  precision  in  the  input  from  device  encoders.  What  was 
originally  thought  to  be  errors  in  the  microcode  of  the  controller  that  cause  it  to  lose  an 
order  of  magnitude  in  reported  precision  are  actually  design  incompatibilities  between  the 
microcontroller  and  the  system  driver  software.  We  have  specified  a  new  design  that  is 
under  development  by  the  manufacturer  of  the  microcontroller,  but  that  was  not 
completed  by  the  end  of  the  project.  Therefore  we  reverted  to  an  earlier  design  using 
only  the  computer  processor.  This  limits  our  haptics  loop  to  only  100  Hz,  but  we  feel  this 
to  be  adequate.  Our  implementation  has  an  alternate  interface  for  the  higher  speed  loop 
provided  in  the  external  processor  should  we  be  able  to  achieve  that  design  in  the  future. 

Instrumentation  of  the  working  channel 

Instrumenting  the  working  channel  of  the  ureteroscope  presented  a  number  of  challenges, 
including  dealing  with  the  size  limitations.  We  worked  to  modify  a  design  for  working 
channel  force  feedback  originally  developed  for  flexible  bronchoscopy  to  fit  within  the 
reduced  size  handle  of  the  ureteroscope.  We  were  unable  to  fit  the  required  mechanism 
in  an  off  the  shelf  ureteroscope,  but  may  experiment  in  the  future  with  a  slightly  larger 
extension  on  the  bottom  of  the  scope.  We  evaluated  working  channel  force  feedback 
using  a  bronchoscope  handle  and  confirmed  that  our  force  model  was  realistic.  We 
implemented  a  new  version  of  the  hardware  interface  layer.  This  creates  a  generic 
framework  that  permits  an  arbitrary  range  and  number  of  hardware  devices  to  be  used, 
enabling  the  use  of  a  range  of  hardware  devices,  including  low-end  devices  in  the  future. 
This  will  make  it  practical  to  substitute  both  the  two-processor  device  as  well  as  an 
operational  ureteroscope  with  working  channel  force  feedback  in  the  future. 

Pathologic/anatomic  variations 

A  major  issue  in  medical  simulation  development  is  the  difficulty  of  producing  new  case 
studies.  We  addressed  this  problem  by  developing  a  set  of  modeling  tools  that  enable  us 
to  quickly  model  a  new  case  from  scratch  or  to  modify  an  existing  case.  This  reduced  our 
turnaround  time  on  production  of  alternative  case  studies  significantly,  and  enabled  a 
much  more  effective  interaction  between  modelers  and  medical  experts.  We  worked  on  a 
method  for  dynamically  inserting  pathologies  into  the  model  but  have  not  been  able  to 
solve  some  low-level  compatibility  problems.  We  have  developed  a  technique  for 
“stitching”  in  a  pathology  or  anatomical  variation  in  a  given  place  in  the  anatomy.  The 
technique  involves  producing  a  polygonal  mesh  model  with  a  set  of  base  or  edge  vertices 
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that  must  be  merged  with  the  underlying  polygonal  anatomical  model.  In  addition  to  the 
base  mesh  where  it  will  be  stitched  into  the  anatomy,  the  pathology  model  includes  a  set 
of  orientation  vectors.  These  include  a  normal  vector  which  after  placement  should  be 
roughly  perpendicular  to  the  average  surface  normal  of  the  local  anatomy  along  with  an 
axial  orientation  vector  after  placement  should  be  approximately  parallel  to  the  axis  of 
local  lumen  of  the  anatomy,  for  example  the  ureter.  We  indicate  an  approximate  area 
where  the  pathology  should  occur,  then  use  a  local  search  procedure  to  find  the  polygon 
that  comes  closest  to  containing  each  vertex.  We  find  the  point  on  that  polygon  that  is 
nearest  the  stitch  vertex  and  from  that  point  place  an  additional  vertex  there,  then  warp 
that  vertex  up  (or  down)  to  the  position  of  the  stitch  vertex.  This  has  the  effect,  for  each 
stitch  vertex,  of  adding  two  new  triangles  to  the  mesh  at  that  point.  The  technique  works 
well,  but  has  some  residual  artifacts.  In  particular,  overall  scaling  of  the  pathology  is 
unrealistically  affected  by  local  anatomical  variations.  We  attempted  to  extend  this  with 
a  technique  to  make  the  pathology  exist  in  it’s  own  reference  frame  whereby  it’s  shape  is 
invariant  even  in  the  presence  of  severe  surrounding  anatomical  movement. 

Better  methods  for  photo-realistic  texture  mapping 

An  issue  in  simulating  long,  tubular  structures  is  that  our  graphics  rendering  hardware 
requires  the  visual  surfaces  to  be  rendered  with  texture  maps  that  are  square,  e.g.  128  x 
128  pixels  or  256  x  256  pixels.  The  resultant  bit  maps,  when  mapped  onto  a  long 
structure,  are  stretched  in  one  dimension  in  a  way  that  makes  it  very  difficult  for  the 
medical  illustrators  to  develop  a  realistic  rendering.  We  developed  a  method  of  using 
smaller  square  bitmaps  and  tiling  them  so  as  to  effectively  allow  the  use  of  rectangular, 
not  square,  textures.  Evaluations  of  the  technique  indicate  that  it  provides  good  results, 
while  simultaneously  minimizing  the  use  of  texture  memory.  Combining  this  with  the 
techniques  reported  above  for  the  endoscopic  lighting  model  and  the  fluid  flow  and 
refraction  models  described  above  provides  an  overall  endoscopic  view  that  is  shown  to 
be  satisfactory  for  ureteroscopy  training.  See  Figures  2  and  3  in  Appendix  C  for 
examples. 

Prototype  Simulator 

Our  progress  of  the  above  mentioned  tasks  has  enabled  us  to  construct  a  prototype 
simulator  (see  Figure  1  in  Appendix  C).  This  simulator  was  demonstrate  at  the  American 
Urological  Associations  Annual  Meeting  (April  30-May  4,  2000)  and  the  World 
Congress  of  Endourology  (September  14-16,  2000).  At  these  meetings,  the  prototype 
simulator  was  used  by  the  leading  experts  in  ureteroscopy  from  around  the  world.  They 
unanimously  agreed  that  the  simulator  was  very  realistic  and  would  be  a  very  valuable 
training  tool.  The  main  request  was  for  the  use  of  working  channel  tools  to  remove  the 
renal  calculi.  While  we  have  the  underlying  technology  for  this,  our  current  resources  do 
not  allow  us  to  complete  this  final  integration  at  this  time.  Efforts  are  currently  underway 
to  secure  funding  to  finish  the  working  channel  tool  functionality  and  to  produce  a  final 
product. 


III.  Key  Research  Accomplishments 

□  Produced  task  analysis  of  flexible  ureteroscopy 

□  Integrated  simulated  fluoroscopic  and  endoscopic  displays 

□  Advanced  contrast  fluid  flow  algorithms 

□  Developed  several  physics-based  models  of  flexible  endoscope 
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□  Constructed  of  an  computer-based  geometric  model  of  the  urinary  tract  from  CT 
data  sets 

□  Developed  set  of  modeling  tools  that  enable  efficient  development  of  anatomic 
and  pathologic  variations 

□  Designed  and  built  devices  that  allow  for  the  application  of  force  feedback  on 
working  channel  tools 

□  Built  prototype  ureteroscopy  simulator 

□  Prototype  simulator  praised  by  experts  in  ureteroscopy  at  the  American 
Urological  Associations  Annual  Meeting  2000  and  the  World  Congress  of 
Endourology  2000  meeting. 


IV.  Reportable  Outcomes 

1 .  Tasto  JL,  Verstreken  K,  Brown  JM,  Bauer  JJ.  PreOp™  Endoscopic  Simulator:  From 
Bronchoscopy  to  Ureteroscopy.  Proceedings  of  Medicine  Meets  Virtual  Reality  2000; 
Newport  Beach,  CA;  January  27-30,  2000;  344-349.  See  appendix  D. 

2.  Tasto  JL,  Bauer  JJ,  Verstreken  K,  Brown  JM,  Merril  GL.  Training  Simulator  for 
Endoscopic  Procedures.  American  Telemedicine  Association  Fifth  Annual  Conference 
and  Exhibit  Showcase;  Phoenix,  A Z;  May  21-24,  2000.  See  appendix  E. 

3.  Preminger  GM,  Auge  BK,  Greenberg  J,  Tasto  J.  Virtual  Reality  in  Endourology 
Training.  Proceedings  of  World  Congress  of  Endourology  2000;  Sao  Paulo,  Brazil; 
September  14-16,  2000. 

V.  Conclusions 

The  research  conducted  during  this  contract  resulted  in  the  development  of  a  realistic  and 
robust  prototype  simulator  for  training  in  ureteroscopy.  The  prototype  was  evaluated  by 
experts  in  ureteroscopy  at  the  American  Urological  Associations  Annual  Meeting  2000 
and  the  World  Congress  of  Endourology  2000  meeting.  These  experts  concluded  that  the 
simulator  was  very  realistic  and  would  be  a  very  valuable  training  tool. 

Future  work  in  this  area  should  concentrate  on  integrating  the  force  feedback  device  into 
the  working  channel  of  the  ureteroscope  and  developing  the  software  to  simulate  a 
variety  of  working  channel  tools  (e.g.,  stone  baskets).  The  technology  developed  for  this 
prototype  should  also  be  leveraged  to  develop  other  endoscopic  and  fluoroscopic 
simulators. 
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Appendix  A:  Ureteroscopy  Task  Analysis 
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Appendix  C:  Ureteroscopy  Simulator  Images 


Figure  1:  Ureteroscopy  Simulator 
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Introduction  to  Flexible  Ureteroscopy  ca«1 
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Figure  2:  Simulated  Ureteroscopic  View  with  Renal  Calculi 
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Figure  3:  Simulated  Fluoroscopic  View  of  Ureteroscope 
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Appendix  D:  MMVR  Paper 


PREOP™  ENDOSCOPY  SIMULATOR: 
FROM  BRONCHOSCOPY  TO 
URETEROSCOPY 
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HT Medical  Systems,  Inc.,  55  W.  Watkins  Mill  Road,  Gaithersburg,  Maryland  20878 
*  Telemedicine  and  Advanced  Technology  Research  Center  (TATRC),  Ft.  Detrick,  Maryland 


Abstract:  The  high  cost  of  virtual  reality  simulators  has  posed  a  major  obstacle  to  the 
widespread  adoption  of  simulators  for  medical  training.  HT  Medical  broke  through  this 
cost  barrier  by  developing  the  PreOp™  Flexible  Bronchoscopy  simulator,  a  realistic 
training  simulation  system  that  integrates  force  feedback,  multimedia,  and  3D  graphics 
on  a  PC.  We  are  currently  extending  the  PreOp  platform  so  that  it  can  simulate  other 
endoscopic  procedures.  This  paper  discusses  our  efforts  to  extend  the  platform  to 
simulate  flexible  sigmoidoscopy  and  ureteroscopy. 


1.  Introduction 

The  high  cost  of  virtual  reality  simulators  has  posed  a  major  obstacle  to  the  widespread 
adoption  of  simulators  for  medical  training.  HT  Medical  broke  through  this  cost  barrier  by 
developing  the  PreOp™  Flexible  Bronchoscopy  simulator,  a  realistic  training  simulation 
system  that  integrates  force  feedback,  multimedia,  and  3D  graphics  on  a  PC.[1]  This 
system,  which  became  a  commercially  available  product  in  the  spring  of  1999,  consists  of 
PC-based  software,  a  proxy  flexible  bronchoscope  and  a  robotic  interface  device  that  tracks 
the  scope  and  provides  force  feedback  to  the  user. 
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Figure  1:  HT  Medical’s  PreOp™  Flexible  Bronchoscopy  Simulator 

We  are  currently  extending  the  PreOp  platform  so  that  it  can  simulate  other  endoscopic 
procedures.  This  will  allow  the  cost  of  the  simulator  to  be  shared  across  departments  at 
medical  schools  and  hospitals,  accelerating  the  introduction  of  the  PreOp  simulator  into  the 
clinical  setting.  This  paper  discusses  our  efforts  to  extend  the  platform  to  simulate  flexible 
sigmoidoscopy  and  ureteroscopy. 


2.  Medical  Analysis 

2.1  Flexible  Sigmoidoscopy 

The  goal  of  flexible  sigmoidoscopy  is  to  inspect  the  colon  for  pathologic  lesions  using 
an  endoscope.  Often  this  is  performed  as  a  screening  exam  for  colorectal  cancer.  There 
are  two  distinct  stages  of  the  exam.  The  first  stage  consists  of  advancing  the 
sigmoidoscope  from  the  anus  to  the  point  of  maximum  insertion.  The  point  of  maximum 
insertion  depends  on  many  factors  such  as  the  scope  length,  the  patient’s  anatomy,  and  the 
level  of  patient  discomfort.  However,  a  60-cm  scope  can  usually  reach  the  splenic  flexure, 
which  is  where  the  transverse  colon  and  the  descending  colon  meet.  The  second  stage  of 
the  procedure  consists  of  carefully  examining  the  mucosal  walls  of  the  colon  as  the  scope  is 
slowly  withdrawn. 

Simulating  flexible  sigmoidoscopy  is  more  challenging  than  bronchoscopy  for  several 
reasons.  Table  1  summarizes  some  of  the  pertinent  differences  between  bronchoscopy  and 
sigmoidoscopy.  Of  particular  note  is  that  the  tracheobronchial  tree  is  relatively  fixed  and 
rigid  compared  to  the  elastic  and  mobile  colon.  These  differences  greatly  increase  the 
computational  time,  which  is  a  paramount  problem  given  the  constraints  of  15-30  frames 
per  second  for  real-time  simulation.  In  addition,  the  sigmoidoscope  is  twice  the  diameter 
of  the  bronchoscope  and  has  additional  controls,  such  as  left/right  tip  control  and  a  two- 
state  air/water  insufflation  button. 


Table  1:  Comparison  of  Significant  Differences  between  Bronchoscopy  vs.  Sigmoidoscopy 


Procedure  Feature 

Bronchoscoov 

Sigmoidoscopy 

Local  Anatomy 

Rigid 

Smooth  Mucosa 

Elastic 

Mucosal  Folds 

Global  Anatomy 

Fixed 

Mobile 
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Complex  branching  pattern 

Single  tube 

Local  Physiological  Motions 

Bronchial  spasm 

Heart/aortic  pulsations 

Colon  spasm 

Global  Physiological  Motions 

Respiratory  motions 

Peristalsis 

Scope  Length 

55  cm 

60  cm 

Scope  Diameter 

6.0  mm 

12.2  mm 

Scope  Tip  Control 

Up/Down 

Up/Down 

Left/Right 

Scope  Buttons 

Suction 

Suction 

Air/Water  insufflation 

2.2  Flexible  Ureteroscopy 

Ureteroscopy  is  performed  to  remove  kidney  stones,  fragment  large  kidney  stones 
(lithotripsy),  biopsy  a  tumor,  or  remove  stents.  The  goal  of  ureteroscopy  is  to  navigate  the 
endoscope  from  the  entrance  of  the  urethra,  through  the  bladder,  into  the  ureter,  and  up  to 
the  calyces  of  the  kidney.  Access  to  the  ureter  is  typically  obtained  by  inserting  a  rigid 
cystoscope  through  the  urethra  in  the  penis  or  vagina  and  into  the  bladder.  A  guidewire  is 
then  navigated  through  the  ureterovesical  junction  (where  the  ureter  empties  into  the 
bladder).  A  balloon  catheter  is  used  to  dilate  the  distal  portion  of  the  ureter  where  it  inserts 
in  the  bladder.  Occasionally  the  proximal  portion  of  the  ureter  must  also  be  dilated.  The 
flexible  ureteroscope  is  then  advanced  over  the  guidewire  and  into  the  ureter.  The  scope 
can  be  advanced  up  to  the  kidneys  and  procedures  performed. 

Ureteroscopy  presents  unique  challenges  over  and  above  bronchoscopy  and 
sigmoidoscopy.  From  a  simulation  standpoint,  ureteroscopy  can  be  thought  of  as  a  hybrid 
between  bronchoscopy,  sigmoidoscopy,  and  angioplasty.  Like  the  tracheobronchial  tree, 
the  calyces  of  the  kidney  branch  extensively,  allowing  for  multiple  paths.  Like 
sigmoidoscopy,  the  ureter  is  a  single  distensible  tube.  Like  angioplasty,  ureteroscopy 
requires  fluoroscopy,  contrast  injection,  balloon  catheters  and  guidewires.  Thus 
ureteroscopy  requires  a  system  that  incorporates  simulation  technology  from  bronchoscopic 
and  sigmoidoscopic  simulation,  as  well  as  from  HT  Medical’s  PreOp™  Endovascular 
Simulator  (shown  in  Figure  2).  We  have  chosen  to  simulate  the  endoscopic  portion  of 
ureteroscopy  first  and  then  add  the  endovascular  features.  Therefore,  this  paper  will  limit 
the  discussion  to  endoscopic  simulation. 
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Figure  2:  HT  Medical’s  PreOp  ™  Endovascular  Simulator 


3.  Software  Design 

The  tasks  for  the  software  can  be  broadly  defined  as  (1)  receive  information  from  the 
user  through  the  software  interface  and  hardware  interface,  (2)  perform  the  necessary 
computations  that  the  user’s  actions  require,  and  (3)  control  the  resulting  visual,  tactile, 
and  auditory  feedback  that  the  user  receives.  For  example,  (1)  the  user  translates  the 
proxy  sigmoidoscope  5  cm;  (2)  the  software  then  computes  the  consequence  of  this 
action  (e.g.,  collision  of  the  scope  with  the  mucosal  wall);  (3)  a  “red-out”  is  displayed  in 
the  endoscopic  view,  the  virtual  patient  says,  “ouch,”  and  the  robotic  interface  device 
provides  a  resistive  force  that  is  felt  by  the  user. 

3.1  Flexible  Sigmoidoscopy 

From  a  computer-modeling  standpoint,  the  most  significant  difference  between 
bronchoscopy  and  sigmoidoscopy  is  that  the  tracheobronchial  tree  is  a  relatively  rigid  and 
fixed  structure,  while  the  colon  is  much  more  elastic  and  mobile.  These  differences  are 
outlined  in  Table  1.  Cartilaginous  rings  and  plates  support  the  bronchi,  whereas  the  colon 
has  no  such  rigid  support.  The  tracheobronchial  tree  does  not  twist,  rotate,  and  loop  like 
the  colon  does. 

The  key  element  in  the  software  design  is  therefore  realistic  modeling  of  the  colon, 
the  flexible  sigmoidoscope,  and  the  colon-scope  interaction.  This  modeling  can  be 
divided  into  global  and  local  modeling  tasks.  Global  modeling  defines  the  interaction 
between  the  entire  scope  and  the  entire  colon  model.  Local  modeling  concentrates  on  the 
interactions  between  the  scope  tip  and  portion  of  the  colon  that  is  located  in  the 
immediate  environment  of  the  scope  tip  (which  is  where  the  scope  view  originates). 

Examples  of  behaviors  handled  at  the  global  level  are  formation  of  loops  in  the  colon 
(e.g.,  alpha  loops  or  N-loops),  changes  in  length  and  diameter  of  the  colon  with  suction 
and  insufflation  of  air,  shortening/lengthening  of  the  colon  due  to  compression/expansion 
of  the  haustral  folds  in  an  accordion-like  fashion  (i.e.  “sleeving  up  the  colon”),  and 
“paradoxical  movement”  of  the  scope  tip  (scope  tip  retracts  as  scope  shaft  is  advanced). 

Examples  of  behaviors  handled  at  the  local  level  are  deformation  of  the  colon  wall  in 
response  to  pressure  from  the  tip  of  the  scope,  “red-out”  of  the  display  when  the  scope  tip 
pushes  against  the  colon  wall,  the  scope’s  interaction  with  mucosal  (haustral)  folds  or 
other  detailed  structures  such  as  polyps,  and  force  feedback  resulting  from  interaction  of 
scope  tip  with  the  colon  wall 

3.2  Flexible  Ureteroscopy 

As  mentioned  above,  flexible  ureteroscopy  requires  the  use  of  several  simulation 
techniques.  Therefore,  we  have  chosen  to  divide  the  approach  into  three  categories,  based 
on  the  specific  challenges  posed  by  the  different  anatomical  structures  to  be  modeled.  For 
the  bladder,  accurate  modeling  of  wire  dynamics  is  necessary  for  the  navigation  of  the 
guidewire  into  the  ureter  and  dilatation  of  the  distal  ureter  with  a  balloon  catheter.  The 
ureter  has  properties  similar  to  the  colon  and  will  thus  be  modeled  using  the  global/local 
approach  described  above  for  flexible  sigmoidoscopy.  The  kidney  requires  an  emphasis  on 
path  finding  because  of  the  branching  pattern  of  the  calyces.  Accurate  collision  detection  is 
necessary  to  constrain  the  scope  to  the  chosen  path. 
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4.  Hardware  Design 

Our  present  endoscopy  simulator  comprises  a  computer,  a  display  monitor,  and  the 
AccuTouch™  Endoscopic  Interface  Device  (patent  pending).  The  interface  device 
consists  of  a  proxy  endoscope  that  looks  like  a  real  endoscope  and  is  electrically 
connected  to  the  second  part  of  the  interface  device,  a  robotic  “mannequin”  into  which 
the  endoscope  can  be  inserted.  Various  sensors,  detailed  below,  track  the  state  of  the 
system  and  send  this  information  back  to  the  host  computer  via  a  data  cable.  The 
simulation  uses  this  state  information  to  compute  appropriate  visual  and  tactile  responses 
to  the  motions  imparted  by  the  user.  These  responses  are  transmitted  to  the  user  through 
visual,  audio  and  haptic  modalities,  creating  a  powerful  illusion  for  the  user  that  he/she  is 
inserting  the  endoscope  into  a  real  patient. 


Figure  3:  PreOp  interface  device 

The  endoscopy  “mannequin”  approximately  represents  the  anatomical  insertion  site, 
accepts  any  of  several  proxy  scopes,  tracks  scope  insertion  and  rotation,  and  provides 
force  feedback  through  the  endoscope  to  the  user's  hands.  Representation  of  the 
anatomical  insertion  site  is  achieved  through  use  of  a  detachable  local  anatomic  model, 
which  is  user-adjustable  into  horizontal,  sideways,  or  vertical  configurations. 

The  robotic  “mannequin”  has  been  designed  to  accept  proxy  bronchoscopes, 
sigmoidoscopes,  colonoscopes,  gastroscopes,  and  ureteroscopes  through  the  appropriate 
anatomic  models.  Tracking  of  scope  motions  is  achieved  with  sensors  that  monitor 
insertion  and  rotation  of  the  scope  tube,  while  electrical  actuators  provide  translational 
and/or  rotational  force  feedback  based  on  the  state  of  the  endoscopy  simulation. 

The  bronchoscope  was  designed  to  look  and  feel  like  a  real  scope,  with  the  same 
controls,  force  feedback,  and  scope  tube  as  the  real  medical  tool.  It  accepts  a  proxy 
working  channel  tool,  tracking  both  tool  translation  and  rotation,  and  provides 
translational  force  feedback  to  the  user.  The  scope  also  tracks  the  motion  of  the  thumb 
lever,  and  monitors  the  state  of  the  suction  button  and  video  control  buttons,  making  that 
information  available  to  the  simulation.  These  sensor  readings  and  actuator  commands 
are  transmitted  to  the  host  computer  via  a  custom  data  cable,  which  looks  and  feels  like 
the  video/light  source  cable  that  normally  is  attached  to  the  scope. 
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Figure  4:  Manipulation  of  working  channel  tool 


4.2  Flexible  Sigmoidoscopy 

We  have  developed  a  sigmoidoscope  following  similar  design  guidelines.  As  with 
the  bronchoscope,  the  sigmoidoscope  accepts  a  proxy  working  channel  tool,  tracking  its 
translation  and  rotation,  and  providing  translational  force  feedback  to  the  user.  The  scope 
tracks  the  two  knobs  that  control  up/down  and  left/right  motion  of  the  scope  tube  tip,  and 
has  locking  mechanisms  as  in  the  real  scope.  In  addition  to  the  suction  and  video  control 
buttons,  the  sigmoidoscope  has  a  two-stage  button  for  controlling  air  insufflation  and 
water  instillation.  These  sensor  readings,  along  with  the  actuator  commands,  are 
transmitted  back  to  the  host  computer  through  the  same  type  of  data  cable  as  with  the 
bronchoscope. 

4.2  Flexible  Ureteroscopy 

Currently  in  development,  the  proxy  ureteroscope  will  incorporate  many  of  the  features 
described  above.  As  in  the  previous  cases,  the  scope  will  need  to  track  its  working  channel, 
along  with  the  control  levers  and  buttons.  Subtleties  of  the  design  include  accommodating 
the  smaller  diameter  scope  tube  (3.2mm,  as  opposed  to  5.3mm  for  the  bronchoscope  and 
12.2mm  for  the  sigmoidoscope)  and  incorporating  the  use  of  guide  wires  and  balloon 
catheters  during  the  procedure. 


5.  Conclusion 

The  PreOp  Endoscopy  Simulator  was  designed  to  be  modular  in  nature  so  that  all 
endoscopic  procedures  could  be  simulated  on  the  same  software  and  hardware  platforms. 
The  first  endoscopic  procedure  simulated  on  the  PreOp  platform  was  flexible 
bronchoscopy.  This  simulator  is  commercially  available  and  has  been  sold  both  in  the 
United  States  and  internationally.  We  are  currently  developing  modules  for  flexible 
sigmoidoscopy  and  flexible  ureteroscopy.  These  additional  modules  will  allow  the  cost  of 
the  system  to  be  shared  across  medical  specialties,  thus  making  the  system  affordable  and 
accelerating  the  use  of  simulation  in  the  medical  setting. 


HT  Medical  Systems,  Inc. 


Final  Report 


Page  23 


6.  Acknowledgments 


This  research  was  funded  in  part  by  the  Department  of  the  Army  under  Cooperative  Agreement  No. 
DAMD 17-99-2-9031.  The  U.S.  Army  Medical  Research  Acquisition  Activity,  820  Chandler  Street,  Fort 
Detrick,  MD  21702-5014  is  the  awarding  and  administering  acquisition  office.  The  content  of  this 
publication  does  not  necessarily  reflect  the  position  or  the  policy  of  the  Government,  and  no  official 
endorsement  should  be  inferred. 


References 

[1]  Bro-Nielsen  M,  Tasto  JL,  Cunningham  RL,  Merril  GL.  PreOp™  Endoscopic  Simulator:  A  PC-based 
immersive  training  system  for  bronchoscopy.  In:  Proceedings  of  Medicine  Meets  Virtual  Reality  7;  San 
Francisco,  CA;  January  20-23,  1999;  76-82. 


HT  Medical  Systems,  Inc. 


Final  Report 


Page  24 


Appendix  E:  ATA  Abstract 


Training  Simulator  for  Endoscopic  Procedures 
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Michael  Brown,  Ph.D.,  Greg  Merril 
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*Telemedicine  and  Advanced  Technology  Research  Center  (TATRC),  Ft.  Detrick, 
Maryland 

Medical  simulators  offer  the  potential  to  improve  medical  training,  decrease  costs,  and 
provide  objective  measurements  of  procedural  competence  for  credentialing  and 
certification.  However,  the  two  biggest  challenges  to  the  adoption  of  this  technology  has 
been  the  high  cost  and  inadequate  realism  associated  with  this  technology.  Until 
recently,  computer-based  simulations  required  expensive  high-end  computers  to  maintain 
the  real-time  frame  rates  necessary  for  realistic  simulation. 

These  challenges  were  overcome  with  the  introduction  of  HT  Medical’s  PreOp™ 
Flexible  Bronchoscopy  simulator,  a  realistic  training  simulation  system  that  integrates 
force  feedback,  multimedia,  and  3D  graphics  on  a  PC.  This  presentation  will  focus  on 
the  research  and  development  efforts  underway  to  extend  this  simulation  system  to  other 
endoscopic  procedures,  such  as  ureteroscopy  and  flexible  sigmoidoscopy.  We  describe 
the  trade-offs  and  solutions  that  we  developed  to  overcome  the  challenges  of  simulating 
the  wide  variety  of  endoscopic  procedures  on  the  same  hardware  and  software  platform. 
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Engineering,  and  Doctor  of  Medicine  degree.  Prior  to  joining  HT  Medical,  Dr.  Tasto  was 
involved  in  innovative  curriculum  development  for  medical  training.  He  has  worked  as 
an  engineer  in  the  field  of  medical  imaging  systems  simulation.  In  addition,  Dr.  Tasto 
developed  simulation  software  while  employed  at  General  Motors.  He  has  published  in 
the  fields  of  medical  simulation  systems,  medical  education,  emergency  medicine,  and 
electrical  engineering. 
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